Eye movement detecting device, electronic device and system

ABSTRACT

According to one embodiment, an eye movement detecting device comprises first, second, third, fourth and fifth electrodes. A line connecting the first and the third electrodes passes through the right eye and a line connecting the second and the fourth electrodes passes through the left eye on at least one of a front view, a plan view or a side view. A distance between the fifth and the first electrodes is equal to a distance between the fifth and the second electrodes. A distance between the fifth and the third electrodes is equal to a distance between the fifth and the fourth electrodes. The detector respectively detects a horizontal movement of the right eye and a horizontal movement of the left eye.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of and claims the benefit of priorityunder 35 U.S.C. § 120 from U.S. application Ser. No. 17/375,097 filedJul. 14, 2021, which is a continuation of U.S. application Ser. No.16/568,333 filed Sep. 12, 2019, which is a division of U.S. applicationSer. No. 16/004,851 filed Jun. 11, 2018 (now U.S. Pat. No. 10,466,781issued Nov. 5, 2019), and claims the benefit of priority under 35 U.S.C.§ 119 from Japanese Patent Application No. 2018-051471 filed Mar. 19,2018, the entire contents of each of which are incorporated herein byreference.

FIELD

Embodiments described herein relate generally to an eye movementdetecting device, an electronic device including the eye movementdetecting device and a system including the electronic device.

BACKGROUND

As one technique of detecting an eye movement, there iselectrooculography (EOG). In this technique, potentials of right eye andleft eye (hereinafter referred as eye potentials) can be detected byattaching electrodes to the skin close to each of the right eye and theleft eye. The eye movement can be detected based at least in part on apattern of variation in the eye potentials.

Conventional eye movement detecting devices cannot detect a movement ofthe right eye and that of the left eye individually. Thus, it cannotdistinguish whether the right eye and the left eye move in the samedirection or in opposite directions.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 is a front view of an example of a glass-type wearable deviceaccording to a first embodiment.

FIG. 2 is a rear top view of the example of the glass-type wearabledevice.

FIG. 3 is a right front view of a user who is wearing the glass-typewearable device.

FIG. 4 is a block diagram showing an example of an electricalconfiguration of the glass-type wearable device.

FIG. 5 shows a first modification to the placement of a neutralelectrode 46.

FIG. 6 shows a second modification to the placement of the neutralelectrode 46.

FIG. 7 shows a third modification to the placement of the neutralelectrode 46.

FIG. 8 shows EOG signals in a state where the user's line of sight is inthe front direction.

FIG. 9 shows an example of variation in a waveform of the EOG signalswhen the right eye and the left eye both move to the left from the statewhere the line of sight is in the front direction.

FIG. 10 shows an example of variation in a waveform of the EOG signalswhen the right eye and the left eye both move to the right from thestate where the line of sight is in the front direction.

FIG. 11 shows an example of variation in a waveform of the EOG signalswhen the right eye and the left eye both move in a direction in which aconvergence angle increases from the state where the line of sight is inthe front direction.

FIG. 12 shows an example of variation in a waveform of the EOG signalswhen the right eye and the left eye both move in a direction in which aconvergence angle decreases from the state where the line of sight is inthe front direction.

FIG. 13 is a chart showing an example of waveforms of the EOG signalsfor various eye movements.

FIG. 14 is a graph showing an example of results of experiment to detectvariation in convergence angle.

FIG. 15 is a front view of an example of a glass-type wearable deviceaccording to a second embodiment.

FIG. 16 is a block diagram showing an example of an electricalconfiguration of a surgery support system including the glass-typewearable device according to the second embodiment.

FIG. 17A, FIG. 17B and FIG. 17C show an example of an operation of theglass-type wearable device according to the second embodiment.

FIG. 18A and FIG. 18B show another example of the operation of theglass-type wearable device according to the second embodiment.

FIG. 19A and FIG. 19B show a still another example of the operation ofthe glass-type wearable device according to the second embodiment.

FIG. 20 is a block diagram showing an example of an electricalconfiguration of a system including a glass-type wearable deviceaccording to a third embodiment.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings. The disclosure of the embodiments is nothing butone example, and the invention is not limited by the descriptions of theembodiments. Modifications that could easily be conceived by a user withordinary skill in the art are included in the scope of the disclosure.To clarify the descriptions, the drawings may show, for example, thesize and shape of each component more schematically than those in theactual aspect. Elements corresponding to each other in the drawings aredenoted by the same reference numeral and their detailed descriptionsmay be omitted.

In general, according to one embodiment, an eye movement detectingdevice includes a first electrode on a right of a right eye, a secondelectrode on a left of a left eye, a third electrode on a left of theright eye, a fourth electrode on a right of the left eye and a fifthelectrode, which are contactable with a head. A line connecting thefirst electrode and the third electrode passes through the right eye. Aline connecting the second electrode and the fourth electrode passesthrough the left eye on at least one of a front view, a plan view or aside view. A distance between the fifth electrode and the firstelectrode is equal to a distance between the fifth electrode and thesecond electrode. A distance between the fifth electrode and the thirdelectrode is equal to a distance between the fifth electrode and thefourth electrode. The eye movement detecting device further includes afirst detector which detects a first eye potential based at least inpart on a difference between the signal from the first electrode and thesignal from the second electrode with a signal from the fifth electrodeas a reference; a second detector which detects a second eye potentialbased at least in part on a difference between the signal from the firstelectrode and the signal from the third electrode with the signal fromthe fifth electrode as a reference; a third detector which detects athird eye potential based at least in part on a difference between thesignal from the second electrode and the signal from the fourthelectrode with the signal from the fifth electrode as a reference; and afourth detector which detects a horizontal movement of the right eye anda horizontal movement of the left eye based at least in part on thefirst eye potential, the second eye potential and the third eyepotential.

The basic information of eyes will be described. The diameter of anadult eye is about 25 mm. The diameter of an eye of a baby immediatelyafter birth is about 17 mm and increases as the baby grows. Theinterpupillary distance of adult males is about 65 mm. Most commercialstereo cameras are therefore manufactured with a distance between lensesof 65 mm. The interpupillary distance of adult females is severalmillimeters shorter than that of adult males. The eye potential isseveral tens of millivolts. The eyes have a positive potential on thecornea side and a negative potential on the retina side. Measuring thesepotentials on the skin surface, a potential difference (referred to asan eye potential) of several hundreds of microvolts appears.

The rotational movement range of eyes (of general adults) is 50° or lessin the left direction and 50° or less in the right direction in theright-and-left movement (also called a horizontal movement), and it is50° or less in the down direction and 30° or less in the up direction inthe up-and-down movement (also called a vertical movement). The anglerange in the up direction of the up-and-down movement, in which a usercan move his or her eyes at his or her will, is narrow. The reason is asfollows. When the eyes are closed, the vertical-direction eye movementrange is shifted in the upward direction due to the “Bell's phenomenon”in which the eyes move upward. Note that the convergence angle (theangle at which the line of sight of the right eye and the line of sightof the left eye intersect) is 20° or less.

First Embodiment

An example of a configuration of an eye movement detecting deviceaccording to an embodiment will be described with reference to FIG. 1 ,FIG. 2 and FIG. 3 . There is a large variety of eye movement detectingdevices. This embodiment is directed to an eyewear-type wearable devicewith the eye movement detecting device. Eyewear includes goggles andglasses (sunglasses are equivalent to glasses) and, in this embodiment,a glass-type wearable device will be described. FIG. 1 is a front viewof the example of a glass-type wearable device. FIG. 2 is a rear topview of the example of the glass-type wearable device. FIG. 3 is a rightfront view of a user who is wearing the glass-type wearable device.

The eye movement includes a vertical movement in which an eye rotates inthe up/down direction about a horizontal axis and a horizontal movementin which an eye rotates in the right/left direction about a verticalaxis. The vertical movement includes a blink, an eye closing, a wink andthe like. The horizontal movement is largely divided into (i) a slow eyemovement in which the right eye and the left eye move unconsciously inthe same direction, (ii) a line-of-sight movement in which the right eyeand the left eye move consciously in the same direction, and (iii) aconvergence/divergence movement in which the right eye and the left eyemove in opposite directions. The convergence means that the line ofsight of the right eye and the line of sight of the left eye intersectand the divergence means that the line of sight of the right eye and theline of sight of the left eye spread. The eye movement is detected basedat least in part on variation in the eye potential. The eye potentialcan be detected by a difference in a voltage between paired electrodesbetween which an eye is interposed. The eye can be interposed betweenthe paired electrodes in any of the horizontal, vertical, back-and-forthand oblique directions. If the eye potential is detected by the pairedelectrodes arranged to sandwich an eye vertically, it is possible todetect a blink, an eye closing, a wink and the like. If the eyepotential is detected by the paired electrodes arranged to sandwich aneye vertically and horizontally, it is possible to detect a blink, aneye closing, a wink, a slow eye movement and a line-of-sight movement.If the eye potential is detected by the paired electrodes arranged tosandwich an eye in a back-and-forth direction and horizontally, it ispossible to detect a slow eye movement, a line-of-sight movement and aconvergence/divergence movement.

[Placement of Electrodes]

The glasses include a right frame 12, a left frame 14 and a bridge 16connecting the frames 12 and 14 together. In FIG. 1 , the glasses areshown from their front and thus the right-side frame in FIG. 1 is theleft frame 14. With a detector that detects an eye potential only,lenses or glasses need not be fit into the right frame 12 and the leftframe 14. If, however, the user regularly wears his or her glasses, heor she can use glasses with lenses whose power are suitable for the userfit into the right frame 12 and the left frame 14, in place of theuser's regular glasses. If the user does not regularly wear the glasses,simple glasses can be fit into the right frame 12 and the left frame 14.Not in the case of the detector that detects an eye potential only, butin the case of a product that detects a line-of-sight movement or aconvergence angle variation based at least in part on an eye potentialand applies a result of the detection, such as a glass-type wearabledevice capable of augmented reality (AR) display, a liquid crystal panelor an organic electro-luminescence panel for AR display can be fit intoat least part of the glasses in the right frame 12 and the left frame14.

In the first embodiment, in order to detect convergence and divergence,electrodes are placed to sandwich each of the right eye and the left eyefrom a back-and-forth direction that is in phase with the right eye andthe left eye (same vector) on the same plane, and electrodes are placedto sandwich each of the right eye and the left eye from a right-and-leftdirection that is in phase opposite to the right eye and the left eye(opposite vector) on the same plane.

To detect an eye potential of the right eye ER, as shown in FIG. 2 , aright temple electrode 32 is provided on the right of the right eye ER,for example, at a portion of the right temple 18 which is to be put on auser's right ear, and a right nose pad electrode 42 is provided on theleft of the right eye ER, for example, on a surface of a right nose pad22 close to a connecting portion of the right frame 12 and the bridge16, the surface being in contact with the user's nose. In the plan view(FIG. 2 is regarded as a plan view), the right temple electrode 32 andthe right nose pad electrode 42 are placed such that a line connectingthese electrodes 32 and 42 passes through the right eye ER.

As shown in the front view (FIG. 1 is regarded as a front view), theright temple electrode 32 is provided on the left of the right eye ER,and the right nose pad electrode 42 (not shown in FIG. 1 ) is providedon the right of the right eye ER. The right temple electrode 32 and theright nose pad electrode 42 are placed such that a line connecting theseelectrodes 32 and 42 passes through the right eye ER. In the front view,the right nose pad electrode 42 (not shown in FIG. 1 ) is provided onthe slightly above the right temple electrode 32.

In the side view (not shown), the right temple electrode 32 is providedbehind the right eye ER. More specifically, the right temple electrode32 is provided on the left of the right eye ER in the right-side viewand it is provided on the right of the right eye ER in the left-sideview. The right nose pad electrode 42 is provided in front of the righteye ER. More specifically, the right nose pad electrode 42 is providedon the right of the right eye ER in the right-side view and it isprovided on the left of the right eye ER in the left-side view. Theright temple electrode 32 and the right nose pad electrode 42 are placedsuch that a line connecting these electrodes 32 and 42 passes throughthe right eye ER.

FIG. 3 shows that the line connecting the right temple electrode 32 andthe right nose pad electrode 42 passes through the right eye ER on thefront view, plan view and side view of a user's head. Note that the lineconnecting the two electrodes 32 and 42 is not limited to the passage ofthe center of the right eye ER but may pass through any portion of theeye. The same is true of the left eye EL though it is hidden by theuser's face.

The right temple electrode 32 and the right nose pad electrode 42, whichdetect an eye potential of the right eye ER, are placed such that theline connecting these electrodes 32 and 42 passes through the right eyeER in the front, plan and side views; however, in at least one of theplan, front and side views, the electrodes 32 and 42 have only to beplaced such that the line passes through the right eye ER.

Similarly, in order to detect an eye potential of the left eye EL, asshown in the plan view (FIG. 2 ), a left nose pad electrode 44 isprovided on the right of the left eye EL, for example, on a surface of aleft nose pad 24 close to a connecting portion of the left frame 14 andthe bridge 16, the surface being in contact with the user's nose, and aleft temple electrode 36 is provided on the left of the left eye EL, forexample, at a portion of the left temple 20 which is to be put on auser's left ear. The left nose pad electrode 44 and the left templeelectrode 36 are placed such that a line connecting these electrodes 44and 36 passes through the left eye EL.

The right temple electrode 32 and the left temple electrode 36 aresymmetric with regard to a straight line orthogonal to the lineconnecting the right frame 12 and the left frame 14 at the midpoint, thestraight line extending to the back of the user's head from the centerof the user's nose.

As shown in the front view (FIG. 1 ), the left nose pad electrode 44(not shown in FIG. 1 ) is provided on the left of the left eye EL andthe left temple electrode 36 is provided on the right of the left eyeEL. The left nose pad electrode 44 and the left temple electrode 36 areplaced such that a line connecting these electrodes 44 and 36 passes theleft eye EL. In the front view, furthermore, the left nose pad electrode44 (not shown in FIG. 1 ) is provided on the slightly above the lefttemple electrode 36.

In the side view (not shown), the left nose pad electrode 44 is providedin front of the left eye EL. More specifically, the left nose padelectrode 44 is provided on the right of the left eye EL in theright-side view and it is provided on the left of the left eye EL in theleft-side view. The left temple electrode 36 is provided behind the lefteye EL. More specifically, the left temple electrode 36 is provided onthe left of the left eye EL in the right-side view and it is provided onthe right of the left eye EL in the left-side view. The left nose padelectrode 44 and the left temple electrode 36 are placed such that aline connecting these electrodes 44 and 36 passes through the left eyeEL.

The right temple electrode 32 is provided on the side surface (which isin contact with the right side of the user's head) and the undersidesurface (which is in contact with the base of the user's right ear) ofthe right temple 18. When the user wears the glasses, the right templeelectrode 32 is brought into contact with a region of the base of theright ear, in which no hard hair grows, by the weight of the temple 18.The left temple electrode 36 is provided on the side surface (which isin contact with the left of the user's head) and the underside surface(which is in contact with the base of the user's left ear) of the lefttemple 20. When the user wears the glasses, the left temple electrode 36is brought into contact with a region of the base of the left ear, inwhich no hard hair grows, by the weight of the temple 20. Accordingly,the right temple electrode 32 and the left temple electrode 36 arebrought into close contact with the skin of the user to allow an eyepotential to be detected accurately.

Note that the left nose pad electrode 44 and the left temple electrode36, which detect an eye potential of the left eye EL, have only to beplaced such that a line connecting these electrodes 44 and 36 passesthrough the left eye EL in at least one of the plan, front and sideviews.

A forehead pad 26 is provided on the inner side of the bridge 16 incontact with the user's forehead. A neutral electrode 46 is provided onthe surface of the forehead pad 26, which is in contact with the user'sforehead. The neutral electrode 46 is an electrode for securing aneutral potential to detect an eye potential, and is in contact with theskin, such as the forehead. The neutral electrode 46 is provided suchthat the distance between the neutral electrode 46 and the right templeelectrode 32 and the distance between the neutral electrode 46 and theleft temple electrode 36 are equal, and the distance between the neutralelectrode 46 and the right nose pad electrode 42 and the distancebetween the neutral electrode 46 and the left nose pad electrode 44 areequal. The neutral electrode 46 is so provided to detect a convergenceangle, which will be described later. The convergence angle is detectedbased at least in part on detection results of eye movements that aresymmetrical viewed from the front of the right eye ER and the front ofthe left eye EL. For example, in an electrocardiograph, a neutralpotential is taken at a portion of the body where the influence of eyemovements can be ignored, such as the end of the right leg. Though thereis a small influence of eye movements, the eye potential influence,which is exerted upon the neutral electrode by the right eye ER and theleft eye EL, can be equalized by taking a neutral potential at a portioninfluenced equally by the right eye ER and the left eye EL, or thecenter of the forehead.

The right temple electrode 32, right nose pad electrode 42, left nosepad electrode 44, left temple electrode 36 and neutral electrode 46 maybe made of foil such as copper, a metal piece, a metal ball such asstainless steel, a conductive silicon rubber sheet or the like. Theelectrodes 32, 42, 44 and 36 are also called EOG electrodes because theyare intended to detect an eye potential, as will be described later.

[EOG Signal]

As shown in FIG. 2 , a processing unit 30 that detects an eye potentialis attached internally or externally to a portion of one of the temples,for example, the right temple 18 close to the right frame 12. A battery34 for the processing unit 30 is attached internally or externally to aportion of the other temple, for example, the left temple 20, which isclose to the frame 12. The processing unit 30 is not designed simply todetect an eye potential but may perform display control in theglass-type wearable device capable of AR display. When the processingunit 30 is provided not inside but outside the glasses, it can beconnected to the processing unit 30 by wireless or by wire and, in thiscase, the battery 34 can be embedded in the processing unit 30. Theprocessing unit 30 can be divided by function into a first processingunit and a second processing unit which are connected to each other bywireless or by wire. Only the first processing unit can be attached tothe glasses to detect a signal from the electrode, and the secondprocessing unit can be provided outside the glasses to detect an eyepotential from the detected signal from the first processing unit andperform control according to a result of the detection. Mobile terminalssuch as smartphones may be used as the second processing unit. Thesecond processing unit is not limited to mobile terminals connecteddirectly to the glasses but may include a server connected thereto via anetwork.

The signal from the right temple electrode 32 is supplied to a negativeterminal (−) of a first analog-to-digital converter 62 of channel 0, andthe signal from the left temple electrode 36 is supplied to a positiveterminal (+) of the first analog-to-digital converter 62. A first EOGsignal ADC Ch0 indicative of a difference between the signal from theright temple electrode 32 and the signal from the left temple electrode36 is output from the first analog-to-digital converter 62. Since theright temple electrode 32 is placed on the right of the right eye ER andthe left eye EL and the left temple electrode 36 is placed on the leftof the right eye ER and the left eye EL, the first EOG signal ADC Ch0represents the horizontal movement of the right eye ER and the left eyeEL.

The signal from the right temple electrode 32 is supplied to a negativeterminal (−) of a second analog-to-digital converter 64 of channel 1,and the signal from the right nose pad electrode 42 is supplied to apositive terminal (+) of the second analog-to-digital converter 64. Asecond EOG signal ADC Ch1 indicative of a difference between the signalfrom the right temple electrode 32 and the signal from the right nosepad electrode 42 is output from the second analog-to-digital converter64. Since the right eye ER is sandwiched vertically and horizontallybetween the right temple electrode 32 and the right nose pad electrode42, the second EOG signal ADC Ch1 represents the horizontal movement andthe vertical movement of the right eye ER.

The signal from the left nose pad electrode 44 is supplied to a positiveterminal (+) of a third analog-to-digital converter 66 of channel 2, andthe signal from the left temple electrode 36 is supplied to a negativeterminal (−) of the second analog-to-digital converter 66. A third EOGsignal ADC Ch2 indicative of a difference between the signal from theleft nose pad electrode 44 and the signal from the left temple electrode36 is output from the third analog-to-digital converter 66. Since theleft eye EL is sandwiched vertically and horizontally between the leftnose pad electrode 44 and the left temple electrode 36, the third EOGsignal ADC Ch2 represents the horizontal movement and vertical movementof the left eye EL.

The horizontal positions of the two electrodes 42 and 32 regarding thesecond EOG signal ADC Ch1 and the horizontal positions of the twoelectrodes 44 and 36 regarding the third EOG signal ADC Ch2 are opposite(the input terminals of the analog-to-digital converter is reversed).Regarding the second analog-to-digital converter 64, the right templeelectrode 32 on the right of the right eye ER is connected to thenegative terminal. Regarding the third analog-to-digital converter 66,the left nose pad electrode 44 on the right of the left eye EL isconnected to the positive terminal. It is thus possible to detectwhether the right eye ER and the left eye EL move horizontally in thesame direction or in opposite directions from the waveforms of thesecond EOG signal ADC Ch1 and third EOG signal ADC Ch2.

Since the voltage signals from the right temple electrode 32, right nosepad electrode 42, left nose pad electrode 44 and left temple electrode36 are faint, the influence of noise is significant. To cancel thenoise, a series circuit of resistors R1 and R2 is connected between thereference analog voltage Vcc (=3.3 V or 5.5 V) of the analog-to-digitalconverters 62, 64 and 66 and the ground (GND). The neutral electrode 46is connected to a connecting point of the resistors R1 and R2. Theresistors R1 and R2 have the same value of, for example, 1 MΩ. Theanalog-to-digital converters 62, 64 and 66 can detect analog voltagesfrom 0 V (ground voltage) to the reference analog voltage Vcc andconvert an input analog voltage to a digital value in a range of 0 V to3.3 V (=Vcc) with the midpoint of a detectable range of, for example, ½voltage of 3.3 V (referred to as a midpoint voltage). The connectingpoint of the resistors R1 and R2 is connected to a midpoint voltageterminal and the neutral electrode 46 is connected to the connectingpoint of the resistors R1 and R2. Thus, the midpoint voltage of theanalog-to-digital converters 62, 64 and 66 becomes the same as thevoltage of the human body (the forehead). Consequently, the midpointvoltage of the analog-to-digital converters 62, 64 and 66 varies withthe voltage of the human body, and noise mixed into the voltage signalsfrom the EOG electrodes 32, 42, 44 and 36 is not mixed into digitalvalues of outputs of the analog-to-digital converters 62, 64 and 66.Accordingly, the S/N ratio of eye potential detection can be improved.

FIG. 4 is a block diagram showing an example of an electricalconfiguration of the eye movement detecting device. The processing unit30 may include the analog-to-digital converters 62, 64 and 66. Theanalog-to-digital converters 62, 64 and 66 can be attached externally tothe processing unit 30.

The signal from the right temple electrode 32 is supplied to thenegative terminal (−) of the first analog-to-digital converter 62, andthe signal from the left temple electrode 36 is supplied to the positiveterminal (+) of the first analog-to-digital converter 62, thusgenerating an EOG signal ADC Ch0 of channel 0. The signal from the righttemple electrode 32 is supplied to the negative terminal (−) of thesecond analog-to-digital converter 64, and the signal from the rightnose pad electrode 42 is supplied to the positive terminal (+) of thesecond analog-to-digital converter 64, thus generating an EOG signal ADCCh1 of channel 1. The signal from the left nose pad electrode 44 issupplied to the positive terminal (+) of the third analog-to-digitalconverter 66, and the signal from the left temple electrode 36 issupplied to the negative terminal (−) of the second analog-to-digitalconverter 64, thus generating an EOG signal ADC Ch2 of channel 2.

The signal from the neutral electrode 46 is supplied to the midpointvoltage terminals of the analog-to-digital converters 62, 64 and 66, andthe midpoint voltages of the analog-to-digital converters 62, 64 and 66are considered the voltage of the human body detected by the neutralelectrode 46.

The EOG signals output from the analog-to-digital converters 62, 64 and66 are supplied to an eye movement detector 75 that detects an eyemovement (also referred to as eye movement). The eye movement detector75 can be configured by hardware or software. In the latter case, a CPU74, a ROM 76 and a RAM 78 are connected to a bus line, and the eyemovement detector 75 is also connected to the bus line. The eye movementdetector 75 is implemented by the CPU 74 that executes programs storedin the ROM 76. A wireless LAN device 80 is also connected to the busline, and the processing unit 30 is connected to a mobile terminal 84such as a smartphone via the wireless LAN device 80. The mobile terminal84 can be connected to a server 88 via a network 86 such as theInternet. The eye movement detector 75 detects an eye potential based atleast in part on the EOG signals output from the analog-to-digitalconverters 62, 64 and 66. From the detected eye potential, the eyemovement detector 75 can detect a horizontal movement(convergence/divergence) of each of the right eye ER and the left eyeEL, a horizontal movement (a line-of-sight movement) of the eyes, and avertical movement (blink, eye closing, etc.). The eye movement detector75 can also estimate various conditions of the user (for example, theuser lacks concentration and is restless, the user is nervous andstressed, the user is tired and cannot concentrate on his or heroperations) from the detected eye movement. The eye movement detector 75estimates other conditions by changing a program to be executed by theCPU 74 what type of eye movement is detected and what type of conditionis estimated. A condition to be detected can also be instructed from themobile terminal 84.

In place of the wireless LAN device 80, a communication device with acommunication system such as ZigBee (registered trademark), BluetoothLow Energy (registered trademark) and Wi-Fi (registered trademark) canbe used. The detection results (eye movement detection results,condition estimation results, etc.) of the eye movement detector 75 canbe stored temporarily in the RAM 78 and then transmitted to the mobileterminal 84 via a communication device such as the wireless LAN device80. Alternatively, the detection results can be transmitted to themobile terminal 84 in real time. The mobile terminal 84 may store thedetection results in a built-in memory (not shown) and transfer them tothe server 88 via the network 86. In accordance with the detectionresults, the mobile terminal 84 may start any processing or store theprocessing results in the built-in memory or transfer the processingresults to the server 88 via the network 86. The server 88 may aggregatethe detection results of the eye movement detector 75 of a number ofwearable devices and the processing results of a number of mobileterminals 84 and analyze what is called big data.

[Modifications to Placement of Electrodes]

FIG. 5 , FIG. 6 and FIG. 7 each show a modification to the placement ofthe neutral electrode 46. In the foregoing description, the right nosepad 22 and the left nose pad 24 are provided separately, but in themodification shown in FIG. 5 , an integrated, inverted V-shaped orinverted U-shaped nose pad 52 is provided. The right nose pad electrode42 is provided on the inner right of the nose pad 52, the left nose padelectrode 44 is provided on the inner left side of the nose pad 52, andthe neutral electrode 46 is provided on the inside of a top portion ofthe inverted V-shaped or inverted U-shaped nose pad 52. The neutralelectrode 46 can thus be provided in contact with the user's foreheadwithout using the forehead pad 26.

In the modification shown in FIG. 6 , too, an integrated, invertedV-shaped or inverted U-shaped nose pad 54 is provided. The nose pad 54differs from the nose pad 52 in that the nose pad 54 expands towards theuser and the nose pad 52 expands downward. The right nose pad electrode42 is provided on the right side of the node pad 54, the left nose padelectrode 44 is provided on the left side thereof, and the neutralelectrode 46 is provided on the center thereof.

In the modifications shown in FIG. 5 and FIG. 6 , a nose pad that islarger than an ordinary one is used. Thus, even though a glass-typewearable device capable of AR display, which is heavier than ordinaryglasses, is used for a long time, its weight does not make the user'snose painful.

In the modification shown in FIG. 7 , the right nose pad 22 and the leftnose pad 24 are used separately, but the forehead pad 26 is unnecessary.In this modification, the right nose pad electrode 42 and a rightneutral electrode 46 a are provided on the surface of the right nose pad22, which is in contact with the user's nose, and the left nose padelectrode 44 and a left neutral electrode 46 b are provided on thesurface of the left nose pad 24, which is in contact with the user'snose. The right neutral electrode 46 a and the left neutral electrode 46b are electrically short-circuited and become equivalent to one neutralelectrode 46.

[Relationship between Line-of-Sight Movement and EOG Signal]

An example of variation in a waveform of each of the EOG signal ADC Ch0output from the analog-to-digital converter 62, EOG signal ADC Ch1output from the analog-to-digital converter 64 and EOG signal ADC Ch2output from the analog-to-digital converter 66 will be described withreference to FIG. 8 to FIG. 12 . It is assumed that the right eye ER andthe left eye EL move horizontally from the state in which the line ofsight is in the front direction.

FIG. 8 shows that the user's line of sight is in the front direction.When the user looks at the infinite far point, the line of sight of theright eye ER and the line of sight of the left eye EL are parallel. Whenthe user looks at the finite far point, the line of sight of the righteye ER and the line of sight of the left eye EL cross at the far point.When the right eye ER and the left eye EL move to the left (the line ofsight of the right eye ER and the line of sight of the left eye EL moveto the left) from this state, as shown in FIG. 9 , the cornea of theright eye ER that is positively charged comes close to the right nosepad electrode 42, and the retina of the right eye ER that is negativelycharged comes close to the right temple electrode 32. Similarly, thecornea of the left eye EL that is positively charged comes close to theleft temple electrode 36, and the retina of the left eye EL that isnegatively charged comes close to the left nose pad electrode 44. Whenthe right eye ER and the left eye EL both move to the right in thisstate, the state returns to that shown in FIG. 8 . Thus, the first EOGsignal ADC Ch0 output from the first analog-to-digital converter 62 towhich the right temple electrode 32 and the left temple electrode 36 areconnected, has a convex waveform (upwardly convex waveform). The secondEOG signal ADC Ch1 output from the second analog-to-digital converter 64to which the right nose pad electrode 42 and the right temple electrode32 are connected, has a convex waveform (upwardly convex waveform). Thethird EOG signal ADC Ch2 output from the third analog-to-digitalconverter 66 to which the left nose pad electrode 44 and the left templeelectrode 36 are connected, has a concave waveform (downwardly convexwaveform).

Since the right eye ER and the left eye EL move in the same direction(move to the left) as described above, the second EOG signal ADC Ch1 andthe third EOG signal ADC Ch2 make waveforms of opposite phases. Thefirst EOG signal ADC Ch0 makes a waveform of the same phase as that ofthe waveform of the second EOG signal ADC Ch1 having the same +/−relationship with regard to the right/left relationship as that of thefirst EOG signal ADC Ch0.

When the right eye ER and the left eye EL both move to the right (theline of sight of the right eye ER and the line of sight of the left eyeEL move to the right), as shown in FIG. 10 , from the state in which theline of sight shown in FIG. 8 is in the front direction, the cornea ofthe right eye ER that is positively charged comes close to the righttemple electrode 32, and the retina of the right eye ER that isnegatively charged comes close to the right nose pad electrode 42.Similarly, the cornea of the left eye EL that is positively chargedcomes close to the left nose pad electrode 44, and the retina of theleft eye EL that is negatively charged comes close to the left templeelectrode 36. When the right eye ER and the left eye EL both move to theleft in this state, the state returns to that shown in FIG. 8 . Thus,the first EOG signal ADC Ch0 output from the first analog-to-digitalconverter 62 to which the right temple electrode 32 and the left templeelectrode 36 are connected, makes a concave waveform (downwardly convexwaveform). The second EOG signal ADC Ch1 output from the secondanalog-to-digital converter 64 to which the right nose pad electrode 42and the right temple electrode 32 are connected, makes a concavewaveform (downwardly convex waveform). The third EOG signal ADC Ch2output from the third analog-to-digital converter 66 to which the leftnose pad electrode 44 and the left temple electrode 36 are connected,makes a convex waveform (upwardly convex waveform).

Since the right eye ER and the left eye EL move in the same direction(move to the right) as described above, the second EOG signal ADC Ch1and the third EOG signal ADC Ch2 make waveforms of opposite phases.However, the phase of the waveforms is opposite to that in the casewhere the right eye ER and the left eye EL both move to the left. Thefirst EOG signal ADC Ch0 makes a waveform of the same phase as that ofthe waveform of the second EOG signal ADC Ch1 having the same +/−relationship with regard to the right/left relationship as that of thefirst EOG signal ADC Ch0. However, the phase of the waveform of thefirst EOG signal ADC Ch0 in the case where the right eye ER and the lefteye EL both move to the right is opposite to that of the waveform of thefirst EOG signal ADC Ch0 in the case where the right eye ER and the lefteye EL both move to the left.

When the right eye ER moves to the left (the line of sight of the righteye ER moves to the left) and the left eye EL moves to the right (theline of sight of the left eye EL moves to the right), as shown in FIG.11 , from the state in which the line of sight shown in FIG. 8 is in thefront direction to cause convergence in which the line of sight of theright eye ER and the line of sight of the left eye EL cross, namely,when both eyes turn inward so that esotropia or “crossed eyes” occurs,the cornea of the right eye ER that is positively charged comes close tothe right nose pad electrode 42, and the retina of the right eye ER thatis negatively charged comes close to the right temple electrode 32.Similarly, the cornea of the left eye EL that is positively chargedcomes close to the left nose pad electrode 44, and the retina of theleft eye EL that is negatively charged comes close to the left templeelectrode 36. When the right eye ER moves to the right and the left eyeEL moves to the left in this state, the state returns to that shown inFIG. 8 . Thus, the first EOG signal ADC Ch0 output from the firstanalog-to-digital converter 62 to which the right temple electrode 32and the left temple electrode 36 are connected, neither varies nor makesa convex or concave waveform. The second EOG signal ADC Ch1 output fromthe second analog-to-digital converter 64 to which the right nose padelectrode 42 and the right temple electrode 32 are connected, makes aconvex waveform (upwardly convex waveform). The third EOG signal ADC Ch2output from the third analog-to-digital converter 66 to which the leftnose pad electrode 44 and the left temple electrode 36 are connected,makes a convex waveform (upwardly convex waveform).

Since the right eye ER and the left eye EL move in opposite directionsas described above, the second EOG signal ADC Ch1 and the third EOGsignal ADC Ch2 make waveforms of the same phase.

When the eye potentials of the right eye ER and the left eye EL are thesame and the absolute values of rotation angles thereof are the same, alevel of the positive terminal (+) and a level of the negative terminal(−) of the analog-to-digital converter 62 vary in the same direction(negative direction) by the same amount. Thus, a difference between thelevel of the positive terminal (+) and the level of the negativeterminal (−) of the analog-to-digital converter 62 does not vary, nordoes the eye potential of the first EOG signal ADC Ch0.

However, in fact, the plane including the nose pad electrode and thetemple electrode is slightly shifted from the central point of the righteye ER or the left eye EL and thus a slight variation occurs in the eyepotential in accordance with the amount of the shift.

When the right eye ER moves to the right (the line of sight of the righteye ER moves to the right) and the left eye EL moves to the left (theline of sight of the left eye EL moves to the left), as shown in FIG. 12, from the state in which the line of sight shown in FIG. 8 is in thefront direction to cause divergence in which the line of sight of theright eye ER and the line of sight of the left eye EL spread, namely,when both eyes turn outward so that exotropia or “wall eyes” occurs, thecornea of the right eye ER that is positively charged comes close to theright temple electrode 32, and the retina of the right eye ER that isnegatively charged comes close to the right nose pad electrode 42.Similarly, the cornea of the left eye EL that is positively chargedcomes close to the left temple electrode 36, and the retina of the lefteye EL that is negatively charged comes close to the left nose padelectrode 44. When the right eye ER moves to the left and the left eyeEL moves to the right in this state, the state returns to that shown inFIG. 8 . Thus, the first EOG signal ADC Ch0 output from the firstanalog-to-digital converter 62 to which the right temple electrode 32and the left temple electrode 36 are connected, neither varies nor makesa convex or concave waveform. The second EOG signal ADC Ch1 output fromthe second analog-to-digital converter 64 to which the right nose padelectrode 42 and the right temple electrode 32 are connected, makes aconcave waveform (downwardly convex waveform). The third EOG signal ADCCh2 output from the third analog-to-digital converter 66 to which theleft nose pad electrode 44 and the left temple electrode 36 areconnected, makes a concave waveform (downwardly convex waveform). Sincethe right eye ER and the left eye EL move in opposite directions asdescribed above, the second EOG signal ADC Ch1 and the third EOG signalADC Ch2 make waveforms of the same phase. However, the phase ofwaveforms of the second EOG signal ADC Ch1 and the third EOG signal ADCCh2 in “wall eyes” is opposite to that of waveforms thereof in “crossedeyes”.

When the eye potentials of the right eye ER and the left eye EL are thesame and the absolute values of rotation angles thereof are the same, alevel of the positive terminal (+) and a level of the negative terminal(−) of the analog-to-digital converter 62 vary in the same direction(positive direction) by the same amount. Thus, a difference between thelevel of the positive terminal (+) and the level of the negativeterminal (−) does not vary, nor does the eye potential of the first EOGsignal ADC Ch0. However, in fact, the plane including the nose padelectrode and the temple electrode is slightly shifted from the centralpoint of the right eye ER or the left eye EL and thus a slight changeoccurs in the eye potential in accordance with the amount of the shift.

FIG. 13 is a chart of eye potentials (EOG signals) illustrating anexample of the relationship between various eye movements and the EOGsignals ADC Ch0, ADC Ch1 and ADC Ch2 generated from theanalog-to-digital converters 62, 64 and 66. In this chart, the verticalaxis represents sample values of the analog-to-digital converters 62, 64and 66 (for example, 3.3 V, 24-bit analog-to-digital converter), and thehorizontal axis represents time.

As shown in FIG. 11 , neither a convex waveform nor a concave waveformappears in the EOG signal ADC Ch0, but a convex waveform appears in theEOG signals ADC Ch1 and ADC Ch2, with the result that the eye movementdetector 75 detects “crossed eyes” in which the line of sight of theright eye ER and the line of sight of the left eye EL are converged. Asshown in FIG. 12 , neither a convex waveform nor a concave waveformappears in the EOG signal ADC Ch0, but a concave waveform appears in theEOG signals ADC Ch1 and ADC Ch2, with the result that the eye movementdetector 75 detects “wall eyes” in which the line of sight of the righteye ER and the line of sight of the left eye EL are diverged. Since theconvergence and divergence differ only in the waveform of the EOGsignals ADC Ch1 and ADC Ch2 as described above, they may be collectivelyreferred to as convergence in the following descriptions. The amplitudeof the waveforms of the EOG signals ADC Ch1 and ADC Ch2 corresponds tothe degree of convergence (i.e. convergence angle) and the degree ofdivergence. As will be described in detail later with reference to FIG.14 , the nearer the distance, the higher the degree of variation of theamplitude; and the farther the distance, the lower the degree thereof.Accordingly, the nearer the distance, the higher the sensitivity ofdetecting of the variation in amplitude; and the farther the distance,the lower the sensitivity thereof.

As shown in FIG. 9 , a convex waveform appears in the EOG signal ADCCh0, a convex waveform appears in the EOG signal ADC Ch1 and a concavewaveform appears in the EOG signal ADC Ch2, with the result that the eyemovement detector 75 detects the left movement of the line of sight.

As shown in FIG. 10 , a concave waveform appears in the EOG signal ADCCh0, a concave waveform appears in the EOG signal ADC Ch1 and a convexwaveform appears in the EOG signal ADC Ch2, with the result that the eyemovement detector 75 detects the right movement of the line of sight.

A first blink #1, second blink #2 and third blink #3 are respectivelydetected by one, two and three convex pulse waveforms of the same phase,which momentarily increase in level and returns to the original level,in the EOG signals ADC Ch1 and ADC Ch2. The eye movement detector 75detects an eye movement in the vertical direction, namely, an eyeclosing by the combination of a convex waveform (upwardly convexwaveform) made when the line of sight is in the upward direction and aconcave waveform (downwardly convex waveform) made when the line ofsight is in the downward direction in the EOG signals ADC Ch1 and ADCCh2. The motion of the eyes caused by a blink and an eye closing in thevertical direction can be detected based at least in part on one of theEOG signals ADC Ch1 and ADC Ch2. In order to detect a blink and an eyeclosing only, an electrode pair need not be provided on each of theright eye ER and the left eye EL but may be provided on only one ofthem.

An example of result of experiment to detect variation in convergenceangle in this embodiment will be described. FIG. 14 shows an example ofvariation in the second EOG signal ADC Ch1 when the intersection of theline of sight of the right eye ER and the line of sight of the left eyeEL varies as a user who is looking at his or her fingertip 10 cm aheadof his or her nose looks at a farther marker, using a test model of theeye potential detector shown in FIG. 1 to FIG. 4 . In this case, theintersection of the lines of sight varies from a near point to a distantpoint and thus the convergence angle decreases. In FIG. 14 , thehorizontal axis represents a distance at which the intersection of thelines of sight moves in the depth direction from a point 10 cm ahead,and the first plot represents EOG amplitude when the intersection of thelines moves from the point 10 cm ahead to a point 20 cm ahead. Note that10 cm is the shortest distance at which the user can gaze stably. Theminimum detected voltage of the EOG signal is 50 μV. In other words, ifthe EOG signal varies 50 μV or more, the eye movement detector 75 candetect variation in the EOG signal and based thereon, detect variationin the intersection of the line of sight, or variation in theconvergence angle. If a level of the EOG signal does not vary 50 μV ormore, the eye movement detector 75 cannot detect variation in the EOGsignal. Note that the minimum detectable voltage of the EOG signal is 50μV in this embodiment though it becomes lower by using an average valueobtained by integrating the measured values. The amplitude of the EOGsignal depends upon the contact resistance of the electrodes. If amaterial whose contact resistance is low is used for the electrodes, theamplitude of the EOG signal increases, as does the minimum detectablevoltage of the EOG signal.

For example, when the convergence angle decreases as a user who islooking at a marker 30 cm ahead of his or her nose looks at a farthermarker, if the EOG amplitude increases 50 μV, the eye movement detector75 detects variation in EOG amplitude. The EOG amplitude obtained byadding 50 μV to the EOG amplitude caused when the user looks at a marker30 cm ahead, corresponds to a marker 40 cm ahead. In other words, whenthe convergence angle decreases as a user who is looking at a marker 30cm ahead of his or her nose looks at a marker 40 cm ahead of his or hernose looks, the eye movement detector 75 can detect variation in EOGamplitude. If variation in EOG amplitude corresponding to a decrease inconvergence angle corresponding to variation of 10 cm can be detected,it can be said that the detection resolution is considerably high. It isseen from FIG. 14 that when the convergence angle decreases as a userwho is looking at a marker 50 cm ahead of his or her nose looks at amarker 65 cm or more ahead of his or her nose looks, variation in EOGamplitude can be detected. When the convergence angle decreases as auser who is looking at a marker 70 cm ahead of his or her nose looks ata marker 140 cm or more ahead of his or her nose looks, variation in EOGamplitude can be detected.

As described above, according to the first embodiment, there is provideda glass-type wearable device including right and left temple electrodes,right and left nose pad electrodes and a neutral electrode placed to beinfluenced equally by the motion of the right eye ER and the left eyeEL. A convergence angle is detected by detecting the horizontal movementof each of the right eye ER and the left eye EL independently. Since theneutral potential of the neutral electrode is considered a midpointpotential of the analog-to-digital converters that sample the EOGsignals generated from the electrodes, the EOG signals are notinfluenced by noise. Therefore, an eye potential is accurately detectedand thus an eye movement is accurately detected. According to the firstembodiment, furthermore, the horizontal movement of both the right eyeER and the left eye EL (horizontal movement of the lines of sight) canbe detected, as can be the vertical movement of both the right eye ERand the left eye EL.

Since the eye potential detector according to the first embodiment makesit possible to detect convergence that is a user's conscious eyemovement, an application example to perform control corresponding to auser's intention can be achieved by performing control corresponding toa result of the detection. For example, in eyewear capable of ARdisplay, the on/off operation of AR display corresponding to thedetection of convergence angle and the adjustment of a display position(distance) of an AR image can be controlled in hands-free mode.

Furthermore, for an operation, variation in convergence angle peculiarto the operation may be required. If a convergence angle variationpattern is compared with a reference pattern, it is possible to check askill level of a user of the wearable device and determine whether he orshe performs the operation accurately.

An application example of convergence angle detection results will bedescribed. The eye potential detector is incorporated into a glass-typewearable device capable of AR display to control AR display based atleast in part on the convergence angle detection result. For example,the detection of convergence angle variation can be applied as afunction select switch. For example, when the user sees a distant placeas shown in FIG. 8 , AR display is turned off. When the user changes hisor her lines of sight to see a near place (convergence state) as shownin FIG. 11 , the on/off state of display can be controlled to display anAR image. Since the display position of the AR image is set at a shortdistance, the eyes are in convergence state while the user is looking atthe AR image and thus the AR display continues. If the user changes hisor her lines of sight to see the distant place, the AR display is turnedoff. Accordingly, the AR display can be controlled as intended by theuser. As a second embodiment, a surgery support glass-type wearabledevice of the application example will be described.

Second Embodiment

FIG. 15 is a front view showing an example of a glass-type wearabledevice 100 according to a second embodiment. The wearable device 100 ofthe second embodiment differs from that of the first embodiment in thatdisplays (for example, an organic electro-luminescence panel and aliquid crystal panel) 102 and 104 for AR display are fit into at leastpart of the right frame 12 and the left frame 14, respectively. Whenaugmented reality (AR) image is not displayed with in dimensions, thedisplays 102 and 104 need not to be fit into the right frame 12 and theleft frame 14. Assume that an AR image is displayed in three dimensions.EOG electrodes are placed in the same manner as in the first embodiment.The wearable device 100, which is capable of AR display, can be appliedto various examples, but an example of a surgery support system will bedescribed as the second embodiment. Doctors who perform surgery andstaff members who assist the doctors wear the wearable device 100.

FIG. 16 is a block diagram showing an example of an electricalconfiguration of a surgery support system including the wearable device100. The processing unit 30 includes a display controller 112 inaddition to the configuration of the first embodiment. The displaycontroller 112 controls AR display of the displays 102 and 104. Thecontrol of AR display includes on/off control of AR display, displayposition (distance) control of AR image (convergence angle control of ARimage), and the like.

In the surgery support system, a plurality of doctors and staff memberswho are involved in the same surgery and thus a plurality of glass-typewearable devices 100 are used. Therefore, a controller 120 such as ahigh-performance personal computer is preferable to the mobile terminal82 of the first embodiment such as a smartphone as a device to beconnected to the processing unit 30. Though not shown, a plurality ofprocessing units 30 are connected to the controller 120. The controller120 includes a vital data memory 124, a support image memory 122 and asupport information memory 126. The eye movement detector 75 may not beprovided in the processing unit 30. The controller 120 may include aneye movement detector 75.

A vital data measurement unit 127 is connected to the controller 120 tomeasure an electrocardiogram, a blood pressure, a pulse and the totalamount of transfused blood for each patient. These vital data items arestored in the vital data memory 124 in the controller 120 for eachpatient. The support image memory 122 stores support images to supportsurgery. When necessary, the support images in a surgery supportdatabase 130 in the server 88 are downloaded to the controller 120 andstored in the support image memory 122. The support information memory126 stores a support text for surgery support. When necessary, thesupport texts in the surgery support database 130 in the server 88 aredownloaded to the controller 120 and stored in the support informationmemory 126.

An example of an operation of the wearable device 100 will be describedwith reference to FIG. 17A to FIG. 17C, FIG. 18A and FIG. 18B. Assumethat the distance from the user's eyes to an affected area under surgery(real world) is about 40 cm. When the user looks at a point between theuser and the affected area (real world), for example, a point about 30cm distant from the user's eyes, the eye movement detector 75 detectsvariation in the convergence angle of the user. As shown in FIG. 11 ,the eye movement detector 75 detects convergence based at least in parton a fact that the waveform of EOG signal ADC Ch0 does not vary and thatthe waveforms of EOG signals ADC Ch1 and ADC Ch2 become upwardly convex.When the eye movement detector 75 detects convergence, it requests an ARimage regarding support for the surgery for the controller 120 andcauses the display controller 112 to display the AR image. As an exampleof the AR image, a vital data window (AR image) is translucentlyoverlapped with the affected area under surgery (real world) as shown inFIG. 17A. The window includes a plurality of pages, and each of thepages displays an electrocardiogram, a blood pressure, a pulse, thetotal amount of transfused blood and the like. The window is limited toa region of the display to prevent the affected area from being hidden.In response to the request from the processing unit 30, the controller120 reads the vital data of the patient from the vital data memory 124and transfers it to the processing unit 30.

If a distance between the right image and the left image of the AR imageis adjusted when the AR image is displayed in three dimensions, thedisplay position (distance) of the AR image can optionally be set. Here,the display position of the vital data window is set in a position about30 cm ahead of the user's eyes, the about 30 cm being equal to adistance at which the eye movement detector 75 detects convergence ofthe lines of sight of the user. When the vital data window is displayed,the user looks at the position about 30 cm ahead of the user's eyes.Therefore, the user can confirm the contents of the vital data windowinstantly and need not move his or her eyes or adjust the convergenceangle of the lines of sight to gaze at the window, which causes noeyestrain.

The pages of the vital data window can be switched automatically everyfixed time period, for example, every one second or switched with user'sintention based at least in part on the eye movement in anotherdirection detected by the eye movement detector 75. If an eye closing offor example, 0.5 seconds or longer is detected, a page of the vital datawindow can be switched. For example, a page can be switched to thefollowing page when the line of sight moves to the right and switched tothe preceding page when the line of sight moves to the left. FIG. 17Bshows an example in which a page of the window is switched.

In the state shown in FIG. 17A and FIG. 17B, when, for example, the userchanges his line of sight to look at the affected area (real world)under surgery (a point about 40 cm more distant), the eye movementdetector 75 detects that the convergence angle decreases (the distanceto the intersection of the lines of sight of both the eyes increases)and causes the display controller 112 to stop the AR display. Therefore,for example, the user observes only the affected area under surgery asshown in FIG. 17C through the right frame 12 and the left frame 14 ofthe glasses.

When, for example, the user wishes to refer to vital data in the stateshown in FIG. 17C, he or she changes his or her lines of sight to lookat his or her nearby point (about 30 cm distant from the user). The eyemovement detector 75 detects that the convergence angle increases (thedistance to the intersection of the lines of sight of both the eyesdecreases) and causes the display controller 112 to make AR display,with the result that a vital data window as shown in FIG. 17A or FIG.17B is displayed.

The determination that the convergence angle increases or decreases isbased at least in part on the relationship between the EOG amplitude andthe movement distance of the intersection of the lines of sight as shownin FIG. 14 . It can be determined that the convergence angle hasincreased or decreased when an EOG voltage decreases or increases by acertain value.

As described above, the user can turn on or off AR display for surgerysupport in hands-free mode during the surgery. This brings about asignificant advantage because the user's hands are busy during thesurgery.

As another example of the AR display, a support image can be displayedas shown in FIG. 18A. The support image may include an image of anexample of the past surgery for a patient who is in similar conditionand an image of other surgery for the patient. The image may include astill image and a moving image. An image of surgery is captured by acamera (not shown), and the captured image is uploaded to the server 88and stored in the surgery support database 130. As still another exampleof the AR display, support information can be displayed as shown in FIG.18B. The support information includes various items of text dataregarding surgery that is being performed.

The display position (distance) of the support information is set closeto the user, like that of the vital data window. However, the displayposition of the support image can be set at a distance of about 40 cmfrom the user, like that of the affected area under surgery (realworld). The user does not see the vital data window or the supportinformation simultaneously with the affected area under surgery. But itis assumed that the user sees the support image simultaneously with theaffected area under surgery to compare them. If the affected area andthe support image differ in display position when they are compared, theuser needs to adjust focus by moving his or her eyes in the horizontaldirection and thus could get eyestrain. For this reason, unlike in theforegoing description, the support image may be displayed when the eyesare in a state as shown in FIG. 8 and turned off when convergence isdetected as shown in FIG. 11 . If the display position of the supportimage is subtly shifted from the affected area under surgery (realworld) when they are compared, a convergence angle detected by the eyemovement detector 75 is subtly changed. If the display controller 112adjusts the display position (convergence angle between the right andleft images) of the support image to decrease the convergence anglevariation detected by the eye movement detector 75, the possibility ofcausing eyestrain can be lowered further.

The switching from the vital data window to the support image andsupport information can be performed by the user based at least in parton an eye movement in another direction detected by the eye movementdetector 75, like the switching of pages of the vital data window.

In the foregoing example, the on/off switching of AR display isperformed by the convergence angle variation of the user; however, thedetected convergence angle can be used for other control and the on/offswitching can be based at least in part on another eye movement. Forexample, when eye closing is detected more than once, the AR display canbe switched between on and off states. The detected convergence anglecan be used to control the display position of the AR image. In otherwords, the distance between the right image and the left image of the ARimage (which is also referred to as a convergence angle of the AR image)is adjusted based at least in part on the distance to the intersectionof the line of sight of the right eye ER and the line of sight of theleft eye EL at the start of the display of the AR image. The convergenceangle of the AR image is adjusted by the display controller 112. Thus,the user need not move his or her eyes in the horizontal direction togaze at the AR display, which causes no eyestrain.

Another application example of the convergence angle variation and thecontrol of AR display will be described with reference to FIG. 19A andFIG. 19B. This example is directed to a pickup operation in a warehouse.During the pickup operation, a user wears the wearable device 100.

The configuration of the processing unit 30 is not shown because it isthe same as that shown in FIG. 16 . The configuration of the controller120 is not shown because it differs only in that a pickup list is usedin place of the vital data, support image and support information. As inthe first embodiment shown in FIG. 4 , the mobile terminal 84 such as asmartphone can be used in place of the controller 120. The mobileterminal 84 may acquire position information. The configuration of theserver 88 is not shown because it differs only in that map informationabout the positions of shelves in the warehouse, a list of goods on theshelves, and a list of goods to be picked up are used in place of thesurgery support data. The pickup list for each user is downloaded to thecontroller 120 from the server 88.

In this application example, too, AR display is first turned off, andthe displays 102 and 104 of the right and left frames 12 and 14 displaynothing and are therefore transparent. The user thus gazes at the insideof the warehouse, as shown in FIG. 19A and FIG. 19B, through the rightframe 12 and the left frame 14. When the mobile terminal 84 as thecontroller 120 can acquire position information, air tags can bedisplayed on goods on each shelf based at least in part on the mapinformation in the server 88 and the acquired position information. Theuser then sees several meters or several tens of meters ahead.

When the user moves his or her right eye ER and left eye EL horizontallyin opposite directions to cause convergence or “crossed eye” as the userwho is looking at a distance point looks at a near point, the eyemovement detector 75 detects the convergence.

In accordance with the detection of the convergence, the processing unit30 requests for the controller 120 a pickup list indicating goods to bepicked up by the user and causes the display controller 112 to start ARdisplay. An example of the pickup list is shown in FIG. 19A and FIG.19B. The display position of the pickup list is set close to the user.When the user moves his or her right eye ER and left eye EL to changefrom a state in which he or she looks at the pickup list to a state inwhich he or she looks at the warehouse that is distant, as shown in FIG.8 , the eye movement detector 75 does not detect the convergence. Thus,it causes the display controller 112 to stop the display of the pickuplist.

Like during the surgery, during the pickup operation, the user's handsare busy. Thus, a significant advantage in which the AR display can beswitched between on and off states in hands-free mode is brought about.

In FIG. 19A and FIG. 19B, the air tag is displayed at all times;however, it can be displayed only when a user looks at a distant targetobject and turned off when the user looks at a nearby target object inanother example. Assume that a user goes out to perform elevatormaintenance in a building. While the user is looking for the targetbuilding on his or her way, he or she is looking at a distant targetobject; thus, an air tag indicative of the name of the building, thedistance, the purpose of the maintenance, etc., is displayed on thetarget building. When the user looks at his or her nearby point to reada map or the like, the air tag may be turned off. When the user arrivesin front of the elevator in the target building and looks at a portionto be inspected, an air tag is displayed on the portion. When the userlooks at his or her nearby point, an AR image of an operation manual,etc., is displayed in place of the air tag. This example can be appliedto a tourist guide. For example, while a tourist is seeing a scene in atourist spot, an air tag concerning the spot can be displayed and whenhe or she looks at his or her nearby point, the air tag can be turnedoff and a detailed tourist guide can be displayed instead.

As another example of making AR display by sensing convergence, it isconsidered that the eye potential detector is incorporated into ARglasses for sports watching. Images of situations of a game are pickedup at various angles and stored in a server as replay images. The ARglasses are connected to the server via a network. Spectators alwayswatch the game and see the distant target object. At that time, the ARdisplay is turned off. When the spectators in bleacher seats of abaseball stadium wish to see an enlarged image of close play on the homeplate, they move their eyes to become cross-eye to start AR display. Inthe AR display, various replay images are downloaded from the server anddisplayed. The spectators can thus enjoy the replay imagesinstantaneously by their eyes' motion.

In the foregoing descriptions, AR display is started when convergence isdetected, and it is stopped when no convergence is detected. Theopposite is true and in other words, AR display can be made when noconvergence is detected, and it can be stopped when convergence isdetected.

Third Embodiment

As another application example of the convergence detection, there isconfirmation of an operation procedure. FIG. 20 is a block diagramshowing an example of an electrical configuration of an operationconfirmation system including a glass-type wearable device. Though ARdisplay is not essential to the operation confirmation, it can be usedto transmit a confirmation result to a user immediately. The processingunit 30 of the third embodiment is the same as that of the secondembodiment shown in FIG. 16 . Instead of the controller 120, a mobileterminal 84 such as a smartphone can be used as in the first embodimentshown in FIG. 4 . The controller 120 includes a line-of-sight movementdetermination unit 148 and a line-of-sight movement pattern memory 142.The line-of-sight movement determination unit 148 determines thedirection of line-of-sight movement (rightward movement, leftwardmovement, intersection of the lines of sight (convergence)) fromvariation pattern of eye movement detected by the eye movement detector75, and stores a line-of-sight movement pattern in the line-of-sightmovement pattern memory 142. Data of the line-of-sight movement patternmemory 142 is uploaded to the server 88 via the network 86.

The server 88 includes a line-of-sight movement standard pattern memory144 and an operation procedure determination unit 146. The operationincludes an operation in which line-of-sight movement peculiar to theoperation is required. For example, in the railroad industry, thestation staff is required to confirm a distant target object and anearby target object in prescribed order by pointing his or her fingerafter a train has departed. In visual inspection of structures such as abridge and a tunnel, too, an inspector is required to move his or herlines of sight in a prescribed pattern because portions to be viewed arepredetermined. The standard pattern memory 144 stores a standard patternof movement of lines of sight peculiar to such an operation. Theoperation procedure determination unit 146 compares the user'slines-of-sight movement pattern uploaded from the controller 120 withthe standard pattern in the standard pattern memory 144 to determinewhether the user performs his or her operation accurately. The standardpattern may be stored in a database (not shown) for each user. Inaccordance with the variation in results of the determination as timepasses, improvement of the user's skill can be estimated. Alternatively,when the operation procedure determination unit 146 determines that theuser does not perform the operation accurately, it may notify thedisplay controller 112 of the processing unit 30 of that effect via thecontroller 120 and display a warning message on the displays 102 and104.

As another example of the line-of-sight movement standard pattern, thereis a car driving pattern. In driver education for driver licenserenewal, a superior driver's confirmation of safety conditions may beintroduced on video. The drivers in the driver education may not onlywatch the video but also follow the superior driver's confirmation. Theline-of-sight movement pattern is detected when the drivers follow thesuperior driver's confirmation. The detected line-of-sight movementpattern of the driver is compared with that of the superior driver todetermine the drivers' skill in confirming safety conditions. In thetransportation industry, too, if a beginner driver learns the sameline-of-sight movement pattern as that of an expert driver, caraccidents can be reduced.

The foregoing descriptions are not limited to glasses but can be appliedto goggles. For example, an electrode can be provided on the front ofthe goggles in place of the node pads, and an electrode can also beprovided on the belt of the goggles in place of the temples.

[Calibration]

In the electrooculography, no specific angle is obtained as theconvergence angle, but only the motion direction of the eyes that isbrought into a convergence state is obtained. In the foregoingembodiments, therefore, convergence is detected and in the applicationexamples, control is performed based at least in part on two states inwhich convergence is detected or not, irrespective of the convergenceangle. However, when there are plural target objects at different knowndistances, the target objects are seen and an eye potential is regularlymeasured for the different known distances. As a result, an eyepotential for the different known distances can be obtained and three ormore convergence states can be recognized. For example, AR display isturned off when a user looks at a distant target object, a first ARimage is displayed when the user looks at a middle-distance targetobject, and a second AR image is displayed when the user looks at anearby target object.

Note that since the contact resistance of the electrodes varies (lowerswith time), the absolute value of the eye potential relative to thedistance varies with time and the absolute value of the distance cannotbe assured for a long time. For example, the contact resistance is highimmediately after the start of use of the electrodes. Thus, theamplitude of the EOG signal is small and the resistance lowers withtime, and the amplitude of the EOG signal increases. If, however, theeye potential relative to the distance is regularly calibrated, thevariation of the eye potential with time can be estimated and theabsolute value of the eye potential relative to the distance can beassured. For example, in an operation with a fixed focal length, such assurgery and desk work and a business operation that assures that atarget object with a known distance is seen at a point in time in theoperation process, the eye potential can regularly be calibrated in theknown state of the distance.

Though the potential of the human body may vary, this variation can becompensated by regular calibration.

Summary of Embodiments

According to the foregoing embodiments, since the horizontal movement ofthe right eye ER and that of the left eye EL can be detectedindependently, convergence that is an intersection of the line of sightof the right eye ER and the line of sight of the left eye EL can bedetected. The convergence is caused not unconsciously but by user'sconscious eye movement. Hands-free control can thus be performedaccording to a user's intention by performing control in accordance withthe detection of the convergence. For example, in a glass-type wearabledevice that makes AR display, the AR display can be started when theuser becomes “crossed eye” state and it can be finished when the userlooks at a distant target object.

The AR display includes monocular display and binocular display. Theforegoing embodiments can also be applied to the monocular display;however, the binocular display brings about the advantage of detectingconvergence of the AR image (an interval between the right eye ER andthe left eye EL). In displaying an AR image in three dimensions forbinocular display, an interval between the right image and the leftimage (which is also referred to as a convergence angle) is optionallyset. If the convergence angle of the lines of sight of the right eye ERand the left eye EL of a user who is seeing the real world and theconvergence angle of the AR image differ from each other, theconvergence angle of the lines of sight of the right eye ER and the lefteye EL has to be adjusted when the real world and the AR image arecompared, which causes eyestrain. If, however, the convergence angle ofthe AR image is set to coincide with that of the lines of sight of theright eye ER and the left eye EL, the user need not move his or herright eye ER and left eye EL to adjust the convergence angle, whichcauses no eyestrain.

Variation in motion of a user's eyes is detected during his or heroperation, and an elapsed-time pattern of the variation is obtained andcompared with the standard pattern. Accordingly, the contents of theoperation can objectively be confirmed. For example, in an operationsuch as maintenance and inspection, when it is necessary to confirm adistant target object and a nearby target object, it is possible toconfirm whether the operation is performed according to the proceduresby detecting convergence during the operation. A result of theconfirmation can be notified to the user as an alarm. The embodimentscan also be applied to finger-pointing confirmation after a train hasdeparted, as well as the maintenance and inspection.

The comparison with the standard pattern can be used for skill inoperation procedures with convergence as well the confirmation of thecontents of the operation. For example, in driver education for driverlicense renewal, a superior driver's confirmation of safety conditionsmay be introduced on video. The drivers in the driver education not onlywatch the video but also follow the superior driver's confirmation. Theline-of-sight movement pattern is detected when the drivers follow thesuperior driver's confirmation. The detected line-of-sight movementpattern of the driver is compared with that of the superior driver todetermine the drivers' skill in confirming safety conditions. Thus, thedrivers in the driver education understand an ideal convergencevariation in detail.

Note that the horizontal movement of eyes is not limited to the motions(convergence) of the right eye ER and the left eye EL in oppositedirections but includes the rightward movement of the line of sight andthe leftward movement of the line of sight in which the right eye ER andthe left eye EL move in the same direction. These movements can becombined with the detection of convergence to perform hands-freecontrol. Nystagmus can be detected based at least in part on therightward movement of the line of sight and the leftward movement of theline of sight, which are caused together with the convergence when auser feels sleepy. The alarm can thus be given to the user who feelssleepy. Hands-free control can be performed by detecting the eyemovement in the vertical direction as well as in the horizontaldirection and combining the eye horizontal and vertical movements.

The present invention is not limited to the foregoing embodimentsthemselves. When the invention is reduced to practice, its structuralelements can be modified and embodied without departing from the spiritof the invention. Furthermore, a variety of inventions can be made byappropriate combinations of the structural elements of the embodiments.For example, some of the structural elements of the embodiments can beomitted. Moreover, the structural elements of different embodiments canbe combined appropriately.

What is claimed is:
 1. A wearable device comprising: a detectorconfigured to detect a convergence angle at which a line of sight of aright eye of a user wearing the wearable device and a line of sight of aleft eye of the user intersect; and a display configured to display anaugmented reality image based on the convergence angle detected by thedetector, wherein the display displays the augmented reality image whenthe convergence angle is smaller than a certain angle; and the displayturns off the augmented reality image when the convergence angle is notsmaller than the certain angle.
 2. A display method for a wearabledevice comprising: a detector configured to detect a convergence angleat which a line of sight of a right eye of a user wearing the wearabledevice and a line of sight of a left eye of the user intersect; and adisplay configured to display an augmented reality image based on theconvergence angle detected by the detector, the method comprising:displaying the augmented reality image when the convergence angle issmaller than a certain angle; and turning off the augmented realityimage when the convergence angle is not smaller than the certain angle.